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ABSTRACT 

This study presents findings from an investigation of the impact of teachers who graduated from a fully 
online master’s degree program with training in pedagogy and a content-specialization in elementary 
reading and literacy (oERL) on reading achievement in a large urban public school system in the 
northwestern United States. The research team used a non-equivalent group design and matched pairs of 
teachers based on degree, grade-level taught, and teaching experience to construct the study on three 
years of student and teacher data. The study consisted of 70 teachers and 3,828 student observations. 
Hierarchical linear modeling was employed to understand the teachers’ effects on student learning over 
time. Results indicate there was a significant positive effect of the oERL on student achievement. 
Broadly, this study is an example of a serious attempt to ascertain the impact of a high demand and fully 
online program on the community where graduates are employed. More narrowly, these results support 
the view that a fully online program aimed at training teachers can provide opportunities for those 
teachers to obtain the pedagogical content knowledge that can positively influence instructional 
effectiveness. 
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I. INTRODUCTION 

Local education agencies (LEAs) are expected under Federal, state, and local legislative and political 
pressure to address a multitude of community, regional, national, and even international needs, 
requirements, and concerns. Under the No Child Left Behind Act [1] which reauthorizes and expands the 
Elementary and Secondary Education Act (ESEA), LEA’s are mandated to provide evidence that student 
achievement goals are met and that teachers are highly qualified; they are further required to respond to 
individual student needs under Individuals with Disabilities Education Act (IDEA-97) amendments of 
1997 [2], Additionally, NCLB requires that LEAs specify the degree student achievement goals are met 
by specific subgroups of students (e.g., race, English language learner status, special education status and 
social economic status). 


Fully addressing these mandates is complex because it must be done while simultaneously addressing the 
educational needs of a student population that “is ethnically, linguistically, and socio-economically 
diverse” [3]. Moreover, the rapidly changing socio-economic landscape is bringing about an educational 
system that is more urban, diverse, and organizationally complex. For instance, ethnic minorities will 
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comprise the majority of students in U.S. public schools by 2035 [3]. In response, LEAs will need to 
recruit and retain a teacher workforce that can enhance or create new student engagement and 
instructional strategies to address the needs of both locally and internationally diverse second language 
learners [4]. Teachers, therefore, will require education and ongoing professional development that 
provides appropriate pedagogical content knowledge and proficiencies (technical, semantic, diagnostic) to 
meet the challenges of a 21 st century classroom [5, 6]. 


To meet these LEA needs, teacher education programs must (a) recruit a diverse and competent cadre of 
teachers that can communicate and prepare socially, linguistically, and economically diverse students to 
compete in a global economy, (b) provide learning experiences that meet the unique certification, 
classroom preparation, and professional development needs of today’s teacher, (c) provide appropriate 
delivery and participation options (face-face, online, etc.), and (d) address the growing demand for 
outcome data that shows how their graduates add value in terms of student performance on standardized 
academic achievement tests. However, they must also be able to meet these requirements within a 
heightened accountability framework that requires empirical evidence based on achievement test scores 
that supports a teacher’s “value-added” to districts, while working with significantly diminished resources 
and uneven assessment capacity within institutions of higher education and LEAs to promote formative 
assessment at the system, district, and teacher levels [7]. Nonetheless, school districts and teacher 
educators are diligently working on strategies to address these requirements. To inform an urban 
university-district collaborative in their attempt to meet the aforementioned requirements, this article 
presents findings from an investigation into the effects on elementary school student reading achievement 
of teachers receiving masters degrees in a completely online advanced graduate teacher education 
program. 

II. BACKGROUND 

A. Online Instructional Strategies Aimed at Addressing the Diverse 
Educational Needs of Teachers 

School districts and universities engaged in teacher education and professional development are grappling 
with how best to address cost, access, efficiency, and recruitment. Online education is viewed as one 
avenue for addressing these needs. As postsecondary institutions continue to adopt at a rapid pace the 
variety of communication and course delivery modes to support instruction and learning [8, 9, 10, 11], 
faculty and administrators are moving parts of classes and programs or sometimes the entire curriculum to 
online learning platforms [3, 12, 13, 14, 15, 16, 17, 18]. 


The move toward online delivery methods has recently extended to teacher education programs. 
Specifically, teacher educators are attempting to provide full or partial undergraduate, post baccalaureate, 
and graduate teacher education programs and course offerings [15, 17, 19, 20, 21], field experience and 
practicum support [19, 22, 23], and teacher professional development and enrichment opportunities [2, 
24, 25, 26, 27, 28]. 


Creating scalable and nimble approaches to meet teacher education and professional development needs 
is especially salient given the continuing national growth in student enrollment, the attendant need to 
address the learning needs of diverse groups (ESL, special education), and demand for support services to 
educate, engage, connect, and encourage beginner and experienced teachers. Borko, Whitcomb, and 
Liston [29], for example, find that professional teacher professional development programs are 
“increasingly turning to ... contemporary, innovative technologies as a way to reach large numbers of 
individuals at costs lower than those associated with the physical presence of professional development 
facilitators” [30, p. 5]. Dede, Ketelhut, Whitehouse, Breit, and McCloskey identify the need for 
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technology-supported teacher learning: 


.. .that can fit with teachers’ busy schedules, that draws on powerful resources often not available 
locally, and that can create an evolutionary path toward providing real-time, ongoing, work- 
embedded support has stimulated the creation of online teacher professional development (oTPD) 
programs [30, p. 9]. 


Despite the burgeoning literature about the impact of online instruction broadly and increasing demand 
for online education in general, and teacher education and professional development in particular, there 
are mixed findings about the efficacy of these modalities, particularly concerning effective participant 
engagement in learning experience, impact on teacher retention, and participant impact on subsequent 
pupil achievement [18, 31, 32], For instance, in discussing online teacher professional development, 
Colgan, Higginson, and Sinclair [32, 33] declared “most of the research that deals with the topic of online 
professional development is limited to statements of vision, opinion, curriculum integration ideas, and 
description of putative benefits ascribed to web and other networks” [33, p. 315], Almost ten years later, 
this statement was echoed in the Journal of Teacher Education following a review of nearly 400 articles 
about online, face-to-face, and hybrid teacher professional development programs when Dede and 
colleagues wrote that the “evidence of effectiveness is often lacking, anecdotal, or based on participant 
surveys completed immediately after the professional development experience” [30, p. 9], They found 
that only 10% met a quality threshold that could be considered empirically sound research (They note 
criteria for assessment is described in Whitehouse, Breit, McCloskey, Ketelhut, &, Dede [34]). The 2009 
study categorized the collected studies around research themes: program design (evaluation of content, 
pedagogical approach, methods of delivery, and best practice); program effectiveness (participant self- 
reported satisfaction and short-term categorical change outcomes); program technical design (effect of 
communication and multimedia on collaboration and building a learning community); and learner 
interactions (quality of participation and efficacy of online communication and collaboration). Of the 40 
studies established as high quality, only a study by Fishman, Marx, Best, and Tal [35] was reported to 
have rigorously examined the relationship between student outcomes and the goals of an online teacher 
professional development program. More recent studies are beginning to show evidence of positive 
student learning and engagement outcomes associated with online programs. , including. For instance, 
the large scale National Survey of Student Engagement report declares that “course management and 
interactive technologies were positively related to student engagement, self-reported learning outcomes, 
and deep approaches to learning” [36, p. 20]. Additionally, these modalities are thought to have a 
positive and significant role in promoting student-faculty interaction and personal and social 
development. 

B. Approaches and Gaps in the Teacher Effects Research Literature 

Teacher effects models are highly valued strategies for determining professional development and teacher 
education outcomes, but there is considerable controversy regarding the contributions of teacher 
background variables to student learning outcomes, and about the specific models used to conduct the 
analyses. Many educational researchers believe the teacher effects literature shows significant deficits in 
terms of coherent theoretical frameworks and sound empirical research findings. For instance, Palardy 
and Rumberger [37] highlight the McCaffrey, Lockwood, Koretz, and Hamilton [38, p. 113] RAND 
report denoting there is “little convincing evidence on the magnitude of a teacher effect or relative 
importance of an aspect of the teachers as a source of variability in student achievement.” Palardy and 
Rumberger [37] also note that Federal legislation [1] has predominately defined highly qualified teachers 
in terms of background characteristics such as intelligence and aptitude tests, education level (e.g., 
bachelor’s degree for elementary and secondary school teachers and subject matter expertise for 
secondary school teachers), state certification (excluding emergency, provisional, or temporary licenses), 
and other credentials. Palardy and Rumberger [37, p. Ill] highlight several studies [39, 40, 41, 42, 43, 
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44, 45, 46, 47, 48, 49] to support the contention that background characteristics and their relationship to 
student learning is “ambiguous.” The researchers further state there is “little scientific evidence that these 
characteristics have a measureable and consistent direct impact on student achievement” [37, p. 112], but 
do contend there is evidence for indirect impacts. For instance, they note the recent Guarino, et al. [50] 
study of kindergarten students that exhibited a significant relationship of coursework in reading 
instruction methods that were positively associated with the use of reading practices deemed by many as 
advantageous for learning. The empirical results of the Palardy and Rumberger [37] study of first grade 
reading and math achievement, however, did ultimately find that reading gains were associated with 
certification, and they also note that many studies show positive relationships for one or more background 
characteristics. For instance, they note the Wayne and Youngs [48] meta-analysis of 21 studies that 
controlled for socio-economic status (SES) and students’ prior achievement and found evidence that 
“college ratings and test scores had consistently positive associations with achievement gains across grade 
levels and participants, there was less support in the literature for the effects of degrees, coursework, and 
certification...” [48, p. 113]. 


In our view, Palardy and Rumberger [37] correctly point out deficits in the research literature by 
questioning coding strategies, omitted variables such as attitudes and practices, and incomplete models. 
However, we contend a well designed research enterprise with a narrower scope can still significantly 
contribute to the extant research literature: 


The usefulness of small, comparison group studies—as well as large correlational studies that use grosser 
measures - is not in the definiteness of their individual findings but in the contribution to a larger body of 
work from which evidence can be triangulated [39, p. 15]. 


Darling-Hammond, Holtzman, Galin, and Vasquez-Heilig [51] make a compelling case for considering 
the effects of teacher background characteristics on student learning gains, even in isolation from attitudes 
and practices if the empirical framework is limited and warrants this approach. For instance, the 2005 
study looked at fourth and fifth grade student achievement gains for six reading and mathematics tests in 
the Houston public schools over a six-year period for Teach for America. The study considered full 
certification (professional or standard certifications) to be a proxy for a defined set of courses that mapped 
to a test of core academic skills, specialized subject matter, and pedagogical knowledge, a reasonable 
assumption under Texas Administrative Codes (Title 19, Part 7, Rule 230.191, 2004). Specifically, they 
highlight it is reasonable to surmise that a teacher with traditional certification has “the ability to manage 
a classroom, design and implement instruction, and work skillfully with students, parents, and other 
professionals” [51, p. 22], The findings indicate that these types of teachers were indeed significantly 
more effective than other teachers in prompting student achievement gains on three of the six measures 
over the period. There was no instance in the study where a certified teacher influenced gain scores less 
than an uncertified teacher and on five of six tests the uncertified teachers had significant negative gains 
(and one non-significant negative gain). The alternative route teachers had non-significant negative gains 
on five of the six tests. 


In many respects, the Darling-Hammond [39] study underpins the research work reported herein by 
providing a methodological framework for conceptualizing teacher effects and supporting the idea that 
this type of research should inform the work of states, school districts, and teacher educators who need to 
develop and expand the reach of academically sound and efficient preparation routes and understand their 
“value-added”. They contend, as we do, the critical need to identify, support, and retain quality urban 
teacher education programs that “have strong records of preparing capable teachers who stay in the city 
schools” [39, p. 23], 
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C. Constructing an Online Elementary Reading and Literacy Program 
(oERL) for Teachers 

Online teacher education and professional development curricula have suffered from a lack of consistent 
and coherent connections between the specific domains of competence (e.g., reading, math, etc.), 
pedagogy, and student learning and cognition in online teacher education and teacher professional 
development curriculum [52, 53, 54] as cited by McCrory, Putnam, & Jansen [20], Teacher education 
more often than not focuses more on generic pedagogy (e.g., constructivist learning frameworks) than on 
the steps a teacher needs to take to effectively integrate pedagogy with disciplinary knowledge in reading, 
math, or science. These “essential tensions” between improving teacher professional practice and 
“intellectual development in the matters of school curriculum” is a persistent dilemma in teacher 
education and professional development programs [52, p. 951] as cited by McCrory, Putnam, & Jansen 
[20]. McCrory, et al. [20] claim these programs are not adequately allocating attention toward the 
facilitation of a teachers’ ongoing growth in intellectual competence relative to a subject area. Supporting 
this claim, Schrader and colleagues [56] also cited the research of Kinzer, Labbo, Leu, and Teale [57] to 
highlight findings from reading teacher self-reports that contend they are lacking exposure relative to 
progressive literacy teacher education and practices. Lor instance, teachers reported receiving very little 
time focused on reading pedagogy and few reported having mentors or established teachers demonstrating 
effective practices. Moreover, there was a general view that the instructional contexts of teachers at that 
time did not appear to reflect attributes of an authentic problem solving context. 


To address these deficits, the online Elementary Reading and Literacy (ERL) program studied herein 
designed the curriculum and program to focus on two key outcomes: (1) enabling the educator to become 
an “expert” decision maker in the field of reading and literature instruction so s/he may effectively 
address the diverse abilities, socioeconomic, and ethnic backgrounds of children [56, p. 318] citing the 
1998 National Research Council’s Committee on the Prevention of Reading Difficulties in Young 
Children), and (2) fostering the scholar-practitioner in all teachers, learning research-based literacy 
theories and strategies, performing research as part of the curriculum, and implementing the findings and 
reflecting on their practice. The curriculum in place at the time of the study was also designed to meet the 
standards of the International Reading Association [58]. Drawn from professional expertise and research, 
the IRA standards identify what teachers need to know and be able to do in order to be competent literacy 
educators. The oERL program is based on current research and practical, research-based classroom 
strategies to increase the achievement and oral reading fluency of all their students, including 
linguistically, culturally, and academically diverse learners. The program subsequently revised its 
curriculum based on the International Reading Association standards [59], designed to identify the 
knowledge and skills teachers need to confidently manage their classroom literacy programs and to 
effectively address the complexities of teaching reading and writing in today’s classroom. Additional 
consideration was also given to the National Reading Panel’s report [60] on the scientific evidence of 
various approaches to teaching children to read. 


The IRA aligned curriculum is delivered within a program structure that combines online course delivery 
and communication, face-to-face interaction with their peers, and optional enrollment in strictly online 
classes. Some oERL students, such as those in this study, attend the university in cohorts where the 
university works to keep them in courses together throughout their program of study. This gives students 
a sense of the cohort community within the university, enabling cohort members to share and collaborate 
on assignments, course materials, and instructional strategics outside the graduate classroom. During the 
study, the program offered an optional hybrid model, a blended environment where cohorts of students 
from the same schools or districts worked together in person, face-to-face, to conduct course-based 
research and discussions. While the students worked face-to-face, their instructors were at a distance. 
Student to instructor communication and assessments was done weekly or more often through email and 
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telephone. The hybrid model was similar to those studied by Bourne, et.al. [61], who found that “The 
studies on blending, combining face-to-face and online methods for learning, offer rich possibilities for 
what many see as the best of both learning modes” [61, p. 9], Although the program became 100% online 
by 2006, some study participants continued to work together in the online classrooms. They enrolled in 
the same classes, collaborated on developing and implementing classroom strategies, and wrote 
assignments together. This approach to online education, though no longer as unique today as during the 
study, encourages students to share their learning with their own students, colleagues, schools, districts, 
and communities. 


Two additional program features warrant mention. First, oERL students have access to an extensive 
electronic university library, as well as additional online media related to their course requirements such 
as Wiki's, podcasts, etc. Students also have access to media and materials outside of their particular 
courses. The library creates a sense of engagement within and outside the classroom. Students use these 
media in their own schools and classrooms. Second, intensive Discussion-Board participation is a 
required element in every course and serves as a foundation for building the scholar-practitioner 
community. Students' postings must include critical analysis of course materials. Responses to colleagues 
are not just "chat", but are evidence-based discussions of colleagues' postings, fostering a collegial 
sharing of ideas and practices. Each element builds the scholar-practitioner community as students learn 
about one another, help each other in professional development, and understand others’ similar 
experiences. Students in the program teach different grade levels, in different schools and districts 
throughout the United States and globally; through the program they share their research-based strategies, 
ideas, and reflections on course materials and strategies, broadening their experience and deepening their 
understanding. 


The oERL’s early adoption of a fully online program model presents a unique opportunity to study the 
effects of online teacher education. At the time of the study, fully online delivery was much less common 
than it is at present, and the program considered this a significant aspect of its methodology. Drawing 
heavily from the work of Bruce [62] and Grisham and Wolsey [63], the International Reading Association 
[64] position paper entitled “New Literacies and 21st Century Technologies” challenges graduate teacher 
education programs and school leaders engaged in professional education to infuse the curriculum with 
the use of technology to “motivate students, bridge the gap between students’ social and academic uses of 
technology, and, in many cases, provide access to technology for their students” (64, p. 3). 
Unfortunately, the paper also finds that research in the area of literacy and technology integration is only 
now emerging. For instance, the paper highlights the recent work of Coiro et al. [65] as significant, but 
notes the “paucity” in evidenced-based approaches and research (65, p. 3). Given the state of the field at 
present, the curriculum of the oERL, and its mode of delivery, the opportunity to review teacher effects 
on achievement for a completely online teacher education program is truly unique. 

III. METHODOLOGY 
A. Data Sources and Sampling Strategy 

The data for the sample was obtained from three main sources: an urban public school district in the 
northwestern United States, the oERL, and Market Data Retrieval (MDR) an education data vendor. First, 
the majority of teacher and student data was provided by the district using standard fields from their 
district data warehouse under a formal Agreement for Conducting District-Approved Research. The 
research team used existing district assessment and teacher data to construct the dataset used for analysis. 
The LEA approved this low-impact approach, but would not allow classroom observations, teacher 
surveys, or additional data gathering due to teacher time considerations. The district-provided data 
included student demographics, teacher demographics, student assessment performance on multiple 
measures from 2004 to 2008, and some teacher experience and education information. The second data 
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source was the partner university, which provided directory information on oERL graduates. Third, MDR, 
a third party information vendor, provided information on degrees earned by TPS teachers. 


To identify teachers who graduated from the oERL program, we identified all teachers by grade-level 
who worked in the school district from 2006 to 2008 and who graduated from the oERL program before 
the fall of 2007. District records were then used to link the teachers to their students in each of the three 
years of data. To establish the comparison group, we matched each oERL graduate to a non-oERL teacher 
in the district who earned a master’s degree, worked in the same grade, and had similar levels of teaching 
experience. When possible, we matched teachers within schools in order to balance the effect of school- 
level factors on student outcomes. This procedure was done separately in each of the three years. Teacher 
attrition and teacher transfers between grades and schools made it impossible to sustain the initial teacher 
matches over three years. 


The purpose of this matching effort was to eliminate, to the extent possible, unobserved differences 
between teachers that may influence their respective students’ performance independent of their actual 
teaching ability. By matching teachers within the same grade and school, we sought to balance many of 
these external contextual factors that have an independent effect on student achievement or teacher 
performance. For example, we may presume that teachers in the same school and grade are participating 
in similar professional learning activities, receiving similar levels of administration support, and dealing 
with similar students and parents. By balancing these factors across our comparison groups of teachers, 
we allow more confidence that differences in student achievement between the groups reflect systematic 
differences in teachers’ instructional effectiveness rather than differences in other factors. 


The decision to restrict the pool of available matches to teachers with existing master’s degrees was also 
aimed at eliminating unobserved differences between the oERL teachers and their comparison group. 
Although there is strong evidence that students receiving instruction from teachers educated at 
competitive undergraduate institutions have higher levels of achievement than students with less qualified 
teachers [66], the extant literature shows there is very mixed to negative evidence that advanced degrees 
make teachers more effective in the classroom [43, 66, 67, 68]. Nevertheless, teachers who pursue 
master’s degrees may share similar characteristics that may influence their effectiveness independently of 
what they actually learn in the degree program. For example, they may have a similar commitment to the 
teaching profession, which also may influence their professional development activities and work habits. 
By restricting the comparison group to only those teachers with master’s degrees, the intention is to 
balance some of those unobserved shared traits and thus more effectively isolate the impact of the online 
program vs. other programs. Given that the average elementary school in our sample had between three 
and four teachers in a grade, we were unable to match each oERL teacher to a comparison group teacher 
in the same school who also held a master’s degree. Consequently, we had to match across schools in this 
manner for 36% of our sample. 

B. Sample Characteristics 

The final sample using the matching strategies outlined above consisted of 31 teachers who graduated 
from the oERL and were teaching in grades one through five in the school district from 2005 to 2008, and 
39 similar TPS teachers. There were a total of 57 oERL graduates employed that worked in the district 
from 2005 to 2008. However, the final sample includes 31 teachers because (a) we were unable to 
properly match teacher credential data or (b) others the teachers were working in the upper-grades 
(middle school or high school). Sixteen of the teachers were matched to comparison group teachers who 
were in the same school; the remaining teachers did not have master’s degree teachers in the same grade 
level, and were therefore matched to teachers in different schools who held master’s degrees. The match 
was conducted separately within each school year in order to account for teacher attrition and grade level 
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transfers resulting in teachers who were present for less than the full three years of the study. This means 
that the total number of comparison group teachers is 39. while only 31 comparison group teachers were 
used within each study year. Data on the comparison group teachers’ master’s degrees were limited to a 
Yes/No indicator of whether they earned a degree. No data were available on the type of degree or the 
name of the degree-granting institution. This essentially makes the study a test of the average effect of 
earning an oERL degree from the university partner relative to the average effect of earning a master’s 
degree from other postsecondary institutions. 


The oERL teacher characteristics were substantially similar to those of the matched comparison group 
teachers. There was the same proportion (and number) of female teachers (83%) and master’s degrees 
held (100%). The groups also had similar experience levels, although the samples are not perfectly 
balanced because after restricting the possible matches for each oERL teacher to the non-oERL teachers 
in the same grade who held master’s degrees, it was not always possible to find a teacher with exactly the 
same level of experience. The oERL teachers in general had less experience than the comparison group 
teachers with 8% having 3-5 years (comparison group=6%), 26% having 6-10 years (comparison 
group=ll%), and 66% having over 20 years (comparison group=82%). 


Table 1: Characteristics in the school district study student sample 


Ethnicity 

oERL 

Comparison Group 

Difference 

Significance 

% Black 

11% 

11% 

0 


% Hispanic 

24% 

19% 

5 

* 

% Asian 

13% 

11% 

2 


% White 

35% 

43% 

-8 

* 

% ESL 

7% 

4% 

3 


% Sp. Ed. 

9% 

10% 

-1 


Student Obs. 

1834 

1994 

-160 



* p<.10, **p<.05, p<.01 


The oERL graduates in the study were linked to a total of 1,774 student observations of DIB ELS fall to 
spring gains for the sample period. These student observations were compared to 1,740 observations of 
the matched teachers. Table 1 displays the characteristics of these students. It reveals that oERL teachers 
served a higher proportion of minority students than the matched comparison group teachers. The 
differences in the proportion of black students and white students in the oERL and comparison groups 
were statistically significant, whereas the other observed differences were not. 

IV. OUTCOME 

The study used a measure of oral reading fluency administered by the district in fall and spring of the 
academic year —the Dynamic Indicators of Basic Early Literacy Skills (DIBELS). DIBELS consists of 
seven short measures of student literacy that are administered individually to students by the teacher. 
Teachers are provided prescribed protocols for administering and scoring the measures so that the 
measures and scores are comparable for all students across grades and over time. 


This study focused on the results of one DIBELS measure that is given to all students in 1st through 5th 
grade: the DIBELS Oral Reading Fluency Measure (ORF). The ORF is designed to monitor students’ 
ability to read aloud fluently. Students are given a short passage that has been calibrated to measure if 
they are meeting their benchmark goals for reading at grade level. They are asked to read the passage 
aloud for one minute. The teacher then scores the students based on the number of correct words per 
minute that they were able to read [69, 70, 71], The ORF is administered in conjunction with the DIBELS 
retell fluency measure (RTF). The RTF is designed as a comprehension check on the ORF. After 
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students finish the ORF measure, they are asked to retell as many words from the ORF passage as they 
can within one minute. According to the DIBELS creators, the general rule of thumb is that the ORF 
score is a strong overall indication of reading comprehension if the retell score is at least 25% of the oral 
reading fluency score. For example, if a student reads 60 words correctly in one minute and is able to use 
at least 15 words to retell the passage, then the ORF is a reliable measure of his/her overall reading 
comprehension skills. We examined the ratio of the RTF to the ORF and found that at all grade-levels, 
the ratio was above 25 percent for oERL students and non-oERL students. 

Table 2: Ratio of Average Spring RTF Score to Average Spring ORF Score for oERL Sample 



ORF Score 

RTF Score 

Ratio 

Grade 1 

62.7 

25.1 

40.1% 

Grade 2 

103.5 

49.3 

47.6% 

Grade 3 

106.1 

39.2 

37.0% 

Grade 4 

121.8 

53.6 

44.0% 

Grade 5 

136.6 

53.6 

39.2% 


This measure was chosen because it was the best district measure available pre-post for all grades of 
interest that is related to the program of interest, and it was the principle district-selected reading measure 
for the primary grades. One advantage of using the DIBELS ORF is that the district administers the test 
three times each year: once in the fall, once in the winter, and once in the spring. This allowed the 
researchers to calculate the growth in oral reading fluency that students make during the school year as 
the difference between a student’s fall test score and spring test score, rather than relying on spring-to- 
spring results as is required when using state assessments. 


There is substantial empirical evidence to suggest that oral reading fluency is a reliable measure of 
students’ overall reading competence, the most salient characteristic of skillful reading [72, 73, 74]. 
Fuchs, et al. [75] found that oral reading fluency is more highly correlated with performance on 
standardized tests of reading comprehension than more direct measures of reading comprehension. The 
specific DIBELS Oral Reading Fluency measure (ORF) has also been shown to have a moderate to strong 
association with state standardized reading assessments [70, 76, 77, 78, 79, 80]. In a study within an 
urban school district, Riedel [81] found that first grade students’ DIBELS ORF subscale score was the 
strongest predictor of all subscales of reading comprehension gains in first grade. A recent study by 
Roehrig et al. [82] also found moderate correlations between third grade students’ ORF scores and 
reading comprehension scores on the FcaT-SS and SaT-10. However, the study also recognizes that there 
is significant debate about the merits of this measure in terms of its relationship to reading comprehension 
as a proxy assessment and its specific assessment approach [83]. In other words, are the dimensions of 
reading measured on the DIBELS such as phonological awareness, alphabetic principle, accuracy and 
fluency with connected text the right dimensions to measure and in which subgroups are they most 
relevant? 

A. Analytic Strategy 

Because sample and variable selection were dictated by available data, we could not fully implement the 
suggested multilevel theoretical framework (e.g., school, teacher-classroom, and student levels) for 
reviewing the variation in “student achievement gains” [37, p. 117]. We therefore employed a multilevel 
model that used fixed school effects and estimated teacher effects. Palardy and Rumberger [37] note that 
assignment at the classroom level is at least reasonable when undertaking a teacher effects analysis 
because: (a) estimating at only the school level can result in aggregation bias and other statistical issues 
and (b) omitting school level variables in a teacher effects analysis is not apt to confound results. In 
Figure 1, we provide a conceptual outline of the relationship of the measures associated with each level 
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and stage. Please note that the solid arrows indicate a potential cause and effect relationship and the 
dotted arrow indicates an association due to aggregation of a measurement (e.g., mean ORF gains). 

B. Statistical Model 

To compare the oERL program to the matched comparison group, we pursued a residual gain score model 
for fall to spring gains in Oral Reading Fluency on the DIB ELS. Specifically, we used a difference score 
outcome for measuring change between the two repeated measures. The initial oral reading fluency score 
was used as a covariate. The difference between the variance and coefficients for this approach in 
comparison to a repeated growth model approach is negligible except for the magnitude of the intercept 
coefficient for the residual gain score approach. However, by adding prior achievement as a covariate at 
student-level, the outcomes have become similar [37]. 


Consistent with Figure 1, our modeling approach took into account the hierarchical structure of the data 
and allowed us to examine the factors that affect student achievement and understand the association 
between the student and teacher achievement gains. Although our focus was on estimating teacher-level 
effects, we included the school fixed effects to (a) allow for the nesting of teachers within schools, (b) to 
restrict our comparisons to variation within schools, and (c) to account for the fact that we could not 
balance the samples such that each oERL teacher had a match within the same school in every case. 

Figure 1: The Relationship of Student Reading Achievement to Student and Teacher Factors 
SOURCE OF 

VARIATION INPUTS OUTPUTS 



Source: The conceptual diagram was adapted from a figure entitled “a multilevel theoretical framework of classroom and school 
effects” found in Palardy & Rumberger [37], 


31 


Journal of Asynchronous Learning Networks, Volume 15: Issue 2 






























Online Teacher Education: Exploring the Impact of a Reading and Literacy Program on Student Learning 


This was a school-level fixed effects model that examined the factors that affect student achievement at 
both the student and teacher levels [84]. The model controlled for student characteristics (ethnicity, 
English as a Second Language status, and Special Education status) to avoid falsely ascribing a difference 
in average gains between oERL and non-oERL teachers to the university’s program, when they are due to 
differences in the characteristics of students the teachers serve. Additionally, an indicator for whether or 
not the teacher is working in a Reading First school was included to account for the supplemental reading 
instruction that accompanies the Reading First funds from Title II. Also, an indicator of whether the 
teacher is working in a school that has a full-time reading instruction facilitator was included to ensure 
that the effect of the oERL teachers was not confounded by the effect of the instructional facilitators. The 
model also included grade effects to control for differences in gains that are due to differences in the 
distribution of students in grades one through five between the samples and year effects to control for 
differences in achievement that are constant across all students in the sample, but vary over time. 


To understand the effectiveness of the oERL and their matched counterparts, we estimated our model 
with three years of student level data using clustered standard errors to account for the nesting of teachers 
and students within classrooms and schools. The mathematical details of the model follow: 


Y i —fto , + t +rpC. t +S D t frOTj fm s t +y +u t +e i 


Where: 


Y it = Spring Oral Reading Fluency score of student i in year t. 

X = 

“ Vector of student level characteristics, including demographic/special program indicators (Black, 
Hispanic, English as a Second Language and Special Education status) and the fall Oral Reading Fluency 
(ORF) score of student i. The fall pre-test allows us to measure the achievement during an entire school 
X c 

year (fall to spring). " is a vector of the average student characteristics in the classroom of student i at 

X s 

time t; " is a vector of the average student characteristics in the school of student i at time t. The 
coefficients of X at all levels (student, class, and school) were held constant across grades and years. 
Student characteristics were included to account for the fact that students are not randomly assigned to 
teachers and consequently differences in the average performance within a classroom is not assumed to be 
exclusively due to the quality of the teacher, but also the underlying differences in students. 

— Is a binary indicator equal to one if the teacher is a graduate of the online master’s program. The 
coefficient 5 is allowed to vary across grades. Given that the program is aimed at early elementary 
literacy instruction, we hypothesized that the program effect will be stronger on 1st and 2nd grade teacher 
effectiveness than in the upper elementary grades. We also ran a model that treats the effect of the oERL 
as constant across grades to provide a sense of the overall difference in ORF achievement between the 
comparison groups. 

T = 

" Is a vector consisting of two measures of teacher experience - the first is a binary indicator equal 
to one if the teacher has fewer than five years of experience. The second is a binary indicator equal to 1 if 
the teacher has 20 or more years of teaching experience. The sample does not include any teachers with 
fewer than 3 years of experience. 


st Is a school fixed effect that captures all unobserved factors that have a constant effect on a 
school’s performance over the three years, but might also vary across each school. For example, the 
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school fixed effect may capture stable differences across schools in the quality of students that feed into 
the schools, or the quality of the administration and/or instructional program. Including the school fixed 
effect also helped us address the fact that we were not able to match each teacher to a support teacher 
within the same grade in the same school who also holds a master’s degree. This approach ensures we do 
not attribute a difference in average ORF achievement between the students of the oERL teachers and 
those of comparison group teachers that is due to differences in the quality of their respective schools, 
rather than their instruction. The inclusion of school fixed effects means that the program effect - i.e. the 
difference in performance between the online master’s and comparison group teachers’ students - is 
estimated by examining the variation within schools. 

Y = 

gt Is a grade fixed effect for student i at time t; the grade fixed effects in part control for systematic 
differences in ORF scores across grades, including those due to the normal deceleration of words-per- 
minute gains as students mature. 

U ' Is a year effect to account for effects on ORF achievement that are constant across all students in 
the sample, but vary over time. 

V. RESULTS 

The results of the multilevel model revealed that the average student ORF predicted gain for oERL 
teachers across all grades was 2.89 words larger than the gain for the matched teacher comparison group 
(please see Table 4 for regression results for the HLM). This difference was statistically significant after 
controlling for student demographics, school reading program characteristics (i.e. Reading First, 
Instructional Facilitator), grade effects, and year effects. Table 3 presents the ORF gains predicted from 
the HLM for the oERL and matched comparison group by grade level. The data also reveal that the 
difference between the two groups was largest in 1st grade with students of oERL teachers showing gains 
in oral reading fluency that were 4.6 words greater on average than the students of the matched 
comparison group. In 2 nd and 5 th grade, the average student gain of the oERL teachers was 3.1 and 3.6 
words greater, but they were not statistically different from the average student gain of the matched 
comparison groups. Very small and non-significant, but positive differences were found in 4 th grade (0.6 
words). The only grade where students of non-oERL teachers averaged larger gains than students of the 
oERL teachers was in 3rd grade, where the matched sample averaged fluency gains of 1.2 more words per 
minute than the students of the oERL teachers. 


Table 3 helps determine the practical importance of the estimated differences. While the estimated 3.6 
word difference between the oERL group and the comparison group in 5 th grade appears small in 
magnitude, it represents all percent increase in the comparison group’s average fall to spring ORF gain 
of 16.2 words. The average reading gain of oERL 1 st grade students was 7.6 percent higher than the 
average gain of the comparison group students. Similarly, the average gain in 2 nd grade was almost 6 
percent higher and the average gain in 4 th grade was 1.1 percent higher. In 3 rd grade, the oERL group 
averaged gains that were 2.4 percent smaller than those of the comparison group. 


Reasons for the 3 rd grade negative effect are unknown. The study was not able to include measures of 
teacher implementation, approaches to instruction, or the role of teacher education in the lesson 
preparation. It may be that some or all of the 3 rd grade teachers did not respond to the program in the same 
way as the teachers in other grades, instructional strategies specific to 3 rd -grade reading instruction were 
not sufficiently addressed by the program, the district reading curriculum differed significantly from the 
primary grades to grade 3, or that the district professional development and approach to reading in 3 ld 
grade was a stronger influence than master degree attainment. Moreover, as with the observed positive 
effects of the oERL program, the negative 3 rd -grade effect may be due to limitations of the research 
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design that include unobserved differences in the students, teachers, and/or schools and comparison 
samples that could only be addressed with randomization. 

Table 3: Predicted Oral Reading Fluency Fall to Spring Gains 



Matched 

Comparison Group 

oERL 

Teachers 

Difference 

Difference as % of Average Control 
Group Fall to Spring Gain 

Grade 1 

60.8 

65.4 

4.6 

7.5% 

Grade 2 

53.7 

56.8 

3.1 

5.7% 

Grade 3 

49.7 

48.5 

-1.2 

-2.4% 

Grade 4 

50.5 

51.0 

0.6 

1.1% 

Grade 5 

32.5 

36.1 

3.6 

11.1% 


Findings suggest that on average students of oERL graduates are making larger gains in oral reading 
fluency than students of comparable non-oERL graduates. Also, recall that a higher proportion of 
comparison group teachers had over 20 years of experience (82 percent vs. 66 percent). This suggests 
that oERL teachers were more effective despite the fact that on average they were less experienced than 
the comparison group teachers. The results reveal that the effect of the oERL program is not constant 
across all elementary grades. Students of 1st, 2nd, 4th, and 5th grade oERL teachers made larger average 
gains than their non-oERL counterparts; in 3 rd grade the comparison group averaged larger gains. 


Table 4: Returns of Online Masters Degree in Elementary Reading and Literacy on 
Oral Reading Fluency Scores 



Constant 

Effect Model 


Grade-varying 
Effect Model 


Program Effect Estimates 





Online Masters Degree 

2.89 

*** 

4.55 

** 

oERL * Grade 2 



-1.48 


oERL * Grade 3 



-5.76 

* 

oERL * Grade 4 



-3.99 


oERL * Grade 5 



-0.94 


Student Characteristics 





Fall Oral Reading Fluency (ORF) 

0.85 

*** 

0.86 

*** 

Black 

-3.04 

*** 

-3.02 

*** 

Hispanic 

-2.05 

* 

-2.10 

* 

ESL 

-15.37 

*** 

-15.33 

*** 

Special Education 

-16.71 

*** 

-16.68 

*** 

T eacher/Classroom 





Characteristics 





% Special Education 

-4.38 


-4.13 


Avg. Fall ORF 

-0.02 


-0.02 


% ESL 

-16.32 


-15.12 


Teaching Experience <=5 Years 

-1.44 


-1.90 
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Teaching Experience >=5 Years 

1.78 


2.05 


School Characteristics 





Reading First 

1.09 


1.02 


Instructional Facilitator 

-7.29 

** 

-7.38 

** 

% Special Education 

47.80 

** 

46.22 

** 

Avg. Fall ORF 

0.03 


0.03 


% ESL 

16.68 


16.55 


Grade Effects 





Grade 2 

-8.20 

*** 

-7.11 

*** 

Grade 3 

-13.97 

*** 

-11.16 

*** 

Grade 4 

-12.68 

*** 

-10.37 

*** 

Grade 5 

-28.83 

*** 

-28.32 

*** 

Constant 

61.46 

*** 

60.83 

*** 

Year Effects 

Yes 


Yes 


School Fixed Effects 

Yes 


Yes 


Obs = 3514 

3514 


3514 


R-Squared = 0.7327 

0.733 


0.716 



* p <.10; **p < .05; ***p <.01 


Table 4 is provided as reference for those with a technical interest in the specification and results of the 
regression model. The table shows the outcome being predicted was the fall to spring gain on the DIB ELS 
ORF. Each factor in the leftmost column of the table was a term used in the analysis because we expected 
it to associate with ORF gains. The numbers in the columns to the right of each factor represent the 
amount of additional gain attributable to that factor when all other factors are held constant. 


The number we are most interested in is that next to the first term, oERL, which is an indicator of whether 
the teacher was an oERL graduate. This value tells us the average effect of the oERL teacher on students’ 
ORF gains when all other factors (race, ESL, special education, Reading First, Instructional Facilitator, 
etc.) are held constant. It is our “treatment effect”. The second column presents the results of a regression 
model where we assume the “oERL effect” on ORF gains is constant across all grades. That is, we 
assume a first grade oERL teacher has the same effect on ORF gains as a fourth-grade oERL teacher, etc. 
This model is used to give us an “average” effect of the oERL teachers over all students in all grades in 
the sample, which we found to be 2.89 additional words per minute. The third column presents the results 
of a regression model where we test to see if the “oERL effect” is different across grades. That is, we test 
to see if the effect of first-grade oERL teachers are statistically different than the effect of second-, third-, 
fourth-, or fifth-grade oERL teachers. In this table, the number for oERL (5.41) is actually the “oERL 
effect” for first-grade teachers. To get the “oERL effect” for the other grades, one needs to add 4.55 to the 
numbers next to the terms: second grade* oERL, third grade*oERL, fourth grade*oERL, fifth 
grade*oERL. For example, to calculate the oERL effect for fifth-grade students, one would add -1.48 to 
4.55 and arrive at 3.07 (or 3.1 rounded). 


Grade-level indicators are included to control for the normal level of gains made within each grade level, 
which differs by grade level for the ORF gains. The numbers presented for second grade, third grade, 
fourth grade, and fifth grade are negative because they are in reference to the average first-grade gain, 
which is captured in the constant term. The constant is the average value in the sample after all other 
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factors are controlled. In this case, it is interpreted as the average gain for first-grade students in the 
comparison group after all other factors are held constant. For example, the constant value of 60.83 
shown in column 3 is the average ORF gain for first-grade students of comparison group teachers, while 
controlling for student factors (ethnicity, ESL status) and school factors (Reading first, instructional 
facilitator). The model also includes year indicators to account for differences in overall performance 
each year. 

VI. SUMMARY/CONCLUSIONS 

Results of this study support the view that the graduates of the oERL program had a significant influence 
on the oral reading fluency of elementary school students in this large urban district. When compared to 
other teacher education program graduates in the same district, the oERL graduates appeared to 
outperform graduates from other programs in all but one grade. Although the study lacks information on 
which comparison group teachers, if any, received their degrees via an online program, it is reasonable to 
hold that these particular findings reflect positively on teachers known to have received their content and 
pedagogical training through an online teacher education program. Given the significant finding for 
teachers in this analysis and their impact on student achievement, LEAs that control the recruitment, 
hiring, and development of teachers might want to give greater consideration to recruits from online 
teacher education programs. It also gives traditional teacher education programs further evidence that it is 
possible to create a standards-based education program that can be delivered fully online and meet the 
diverse access and learning needs of schools, teachers, and students. 


The study has two potential threats to validity: differences in program emphasis between the treatment 
and comparison groups, and potential self selection bias. Literacy instruction is a key responsibility of 
primary and elementary school teachers, and the extent to which teachers succeed in teaching literacy is 
the outcome studied. The study does not seek to compare the oERL program to other elementary literacy 
programs; it seeks to compare the effects oERL program graduates to other degree program graduates. 
Because of this, the teacher’s master’s degree is not an attribute of the teacher that we want to make 
exactly comparable for the treatment group (oERL teachers) and comparison group (non-oERL teachers). 
Rather, it is the treatment, or intervention that we are evaluating to determine if it associates with 
increased teacher effectiveness, as measured by student reading achievement. To evaluate the 
effectiveness of the oERL, we needed to specify a comparison group with which to compare it. The 
comparison group we chose could be driven by any one of the following questions: (1) does the oERL 
program perform better than the average of all the other teacher education programs used in the field? (2) 
does the oERL program perform better than those teacher education programs that are most commonly 
experienced? (3) does the oERL perform better than other similar teacher education programs? However, 
this study sought to find evidence to answer the first question: Does the oERL program make teachers 
more effective at reading instruction than the average of other master’s degree programs given we did not 
have information on the institutions attended for comparison group participants? 


The goal of early-stage research, such as this, should be to compare the experience (treatment) to a very 
different control conditions to see if you can identify a main effect of the program. We have made the case 
in the literature review that the combination of modality for delivery and content-approach for the oERL 
likely makes this program unique. Once a main effect has been established, it makes sense to proceed to 
comparing the oERL program to similar other programs—an effort that will likely require larger sample 
sizes in order to identify statistically significant differences and access to data to allow the researchers to 
better identify relevant covariates. 


While it is not necessarily a problem that treatment teachers received a degree specializing in elementary 
reading and literacy and control teachers may not have, it may be a problem if teachers who sought out 


36 


Journal of Asynchronous Learning Networks , Volume 15: Issue 2 



Online Teacher Education: Exploring the Impact of a Reading and Literacy Program on Student Learning 


specialized reading and literacy programs were also more interested in reading instruction and motivated 
to improve their reading instruction. If this is the case, these teachers may be more effective reading 
instructors regardless of whether they attend the partner university’s oERL program. This is the primary 
threat to validity that stems from self-selection into treatment groups. This threat can only be overcome 
via a stronger experimental or quasi-experimental design than presented here, thus we do not argue that 
there is a causal relationship between the oERL program and instructional effectiveness. 


Limitations on the research design clearly would not allow for a claim of causation between the 
completion of the oERL degree and teaching effectiveness. However, the finding of a positive effect of 
the oERL program on student achievement provides suggestive evidence that the program may indeed 
improve the effectiveness of elementary literacy instruction. These findings help establish the empirical 
basis for subsequent experimental and quasi-experimental investigations of the efficacy of oERL 
programs. We recognize that since the inception of this particular research project useful roadmaps or 
guiding principles for carrying out research pertaining to online teacher education and professional 
development programs [30] and teacher effects analysis [37] have been published. Although we did not 
cover all of the requirements set forth by these researchers, we recommend that other researchers 
carefully consider them in conceptualizing future studies. 


Additional analyses of the core content areas and tasks covered by the oERL such as those outlined in the 
International Reading Association [85], while beyond the scope of this study, would help in 
understanding which pedagogical content areas have the most significant impact on achievement. It 
would also be useful to replicate Palardy and Rumberger’s [37] consideration of attitudes and practices, or 
Konold, et al’s [7] attempts to understand teacher education program influences on teacher behaviors that 
affect student learning, including “whether these teachers are likely to change their behaviors when 
confronted with pupils not learning [37, p. 310]. Studies in progress by the authors and the oERL program 
analyze changes in teacher effectiveness pre and post degree attainment, using teacher influence on 
student reading scores as the outcome measure, and will enable consideration of the effects of curricular 
changes on teaching and learning. 


Nonetheless, we believe “the usefulness” of this small, comparison group study to be that it contributes to 
the larger body of work from which evidence can be triangulated [6, p. 15] and pragmatically, that it 
supports the view that the type of educational background a teacher brings to the classroom is associated 
with student achievement. Such evidence should continue to prompt LEAs and teacher educators to 
continue to propose studies that seriously consider background characteristics in their research designs 
and look closely at the impact of teacher education curriculum on the instructional approaches of teachers 
and the relationship of these instructional approaches to student learning. 
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